This sample project shows how to connect to multiple AI APIs — Gemini Live, OpenAI Realtime, Gemini image generation, OpenAI DALL-E, and Snap3D — using the Remote Service Gateway package in Lens Studio. It demonstrates real-time audio and video streaming to large language models, text-to-image generation, and text-to-3D generation with interactive results in the Specs environment. See more at the Remote Service Gateway documentation.
NOTE: This project will only work for the Specs platform. You must set the simulation mode in the Lens Studio Preview panel to
Specs (2024). You must also provide your own Remote Service Gateway Key to use the functionality provided by this project.
Designing Lenses for Specs offers all-new possibilities to rethink user interaction with digital spaces and the physical world. Get started using our Design Guidelines
- Lens Studio: v5.15.4+
Note: Ensure Lens Studio is compatible with Specs for your Specs device and OS versions.
- Specs OS Version: v5.64+
- Specs App iOS: v0.64+
- Specs App Android: v0.64+
To update your Specs device and mobile app, please refer to this guide.
You can download the latest version of Lens Studio from here.
The camera feature requires you to use Experimental APIs. Please see Experimental APIs for more details here.
To obtain the project folder, clone the repository.
IMPORTANT: This project uses Git Large Files Support (LFS). Downloading a zip file using the green button on GitHub will not work. You must clone the project with a version of git that has LFS. You can download Git LFS here.
The project should be pre-configured to get you started without any additional steps. However, if you encounter issues in the Logger Panel, please ensure your Lens Studio environment is set up for Specs.
To enable AI features, you must provide a Remote Service Gateway token:
- Install the Remote Service Gateway Token Generator plug-in from the Asset Browser.
- Go to Window > Remote Service Gateway Token.
- Click Generate Token.
- Copy and paste the token into the
RemoteServiceGatewayCredentialsobject in the Inspector.
AIAssistantUIBridge.ts - Connects AI assistants (Gemini Live and OpenAI Realtime) to the Sphere Controller UI and the Snap3D factory.
GeminiAssistant.ts - Manages the Gemini Live WebSocket session, streaming audio and video input and handling text or function call responses.
OpenAIAssistant.ts - Manages the OpenAI Realtime WebSocket session, handling audio streaming, text responses, and function calls with call ID tracking.
ImageGenerator.ts - Wraps the Gemini and OpenAI image generation APIs to return a Texture promise from a text prompt.
InteractableImageGenerator.ts - Connects ASR voice queries to the ImageGenerator and displays the resulting image on screen.
InteractableSnap3DGenerator.ts - Connects ASR voice queries to the Snap3D factory to generate 3D objects at a target scene position.
Snap3DInteractableFactory.ts - Creates interactable Snap3D scene objects from text prompts and manages the full generation lifecycle.
Snap3DInteractable.ts - Receives image preview and final mesh data from the factory and renders them on an interactable scene object.
ASRQueryController.ts - Implements a single-query voice button using ASR that fires an event with the transcribed text on completion.
SphereController.ts - Manages the orb UI element that follows the user's hand, can be placed in world space, and falls back to screen space when out of view.
InternetAvailabilityPopUp.ts - Shows or animates away a popup to inform the user when internet connectivity is unavailable.
APIKeyHint.ts - Displays a hint message when any required API key has not been configured in the Remote Service Gateway Credentials component.
- Open the Preview panel in Lens Studio and set the simulation mode to
Specs (2024). - Ensure your Remote Service Gateway token is correctly set in the
RemoteServiceGatewayCredentialsobject. - Select either the Gemini Live or OpenAI Realtime button in the preview to start a session.
- Use the on-screen orb to activate and deactivate the microphone input to the assistant.
- Build and deploy the project to your Specs device.
- Follow the Specs guide for device testing.
- Look down at your left hand and pinch the orb to activate the AI assistant.
- Speak a prompt — the assistant will respond with voice and text output.
If you have any questions or need assistance, please don't hesitate to reach out. Our community is here to help, and you can connect with us and ask for support here. We look forward to hearing from you and are excited to assist you on your journey!
Feel free to provide improvements or suggestions or directly contributing via merge request. By sharing insights, you help everyone else build better Lenses.
Built with <3 by the Specs team